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ABSTRACT 


In  fiscal  year  1994,  the  federal  government  obligated  more  than  $23.5  billion  towards 
information  technology  (IT)  products  and  services,  which  represents  about  five  percent  of 
its  total  discretionary  spending.  The  Clinger-Cohen  Act  of  1996  was  enacted  by  the  U.S. 
Congress  to  help  prevent  wasteful  government  spending  on  IT  projects  by  mandating  that 
federal  agencies  develop  a  process  to  manage  their  IT  projects  as  investments,  including 
methods  to  continuously  improve  the  efficiency  and  effectiveness  of  their  management 
processes.  The  goal  of  this  study  was  to  develop  a  model  to  assess  a  government 
agency’s  IT  management  processes,  specifically  the  selection,  control,  and  evaluation  of 
IT  investments.  In  order  to  accomplish  this,  various  General  Accounting  Office  (GAO) 
reports  were  reviewed  to  determine  the  best  practices  being  used  to  manage  IT 
investments.  Also,  a  model  was  developed  that  can  be  applied  to  an  agency  that  already 
has  some  IT  management  processes  in  place.  The  major  finding  was  that,  while  the 
critical  processes  and  attributes  identified  by  GAO  are  helpful  in  implementing 
improvements  to  an  agency’s  processes,  each  individual  agency  needs  to  carefully  assess 
the  environment  in  which  it  operates  and  choose  the  management  tools  and  techniques 
that  best  fit  the  agency’s  vision  and  mission  and  its  specific  environment. 
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L  INTRODUCTION 

A.  BACKGROUND 

In  fiscal  year  1994,  the  federal  government  obligated  more  than  $23.5  billion 
towards  information  technology  (IT)  products  and  services.  This  amounted  to  about  five 
percent  of  the  government’s  total  discretionary  spending  for  the  year  [Ref  l:p.2].  In 
contrast  to  some  private  corporations  that  achieved  significant  performance 
improvements  and  costs  savings  with  investments  in  IT,  government  investments  have 
regularly  cost  millions  more  than  expected,  taken  longer  to  deliver,  and  failed  to  deliver 
significant  improvements  in  cost,  speed,  or  quality  of  service. 

In  an  effort  to  ensure  government  agencies  use  their  IT  dollars  wisely.  Congress 
enacted  substantial  legislation  in  the  last  decade.  The  Government  Performance  and 
Results  Act  of  1993  (GPRA)  and  the  Information  Technology  Management  Reform  Act 
of  1996  (later  renamed  the  Clinger-Cohen  Act)  both  aim  to  improve  the  long-term 
planning  process  as  well  as  the  day-to-day  operations  of  executive  agencies. 

GPRA  was  designed  to  improve  federal  business  management,  program 
effectiveness  and  public  confidence  in  government  by  making  government  ofiBces 
develop  measurable  and  attainable  goals  and  performance  measures  that  could  be  used  to 
measure  their  progress.  This  law  mandated  the  development  of  strategic  plans  and  annual 
target  goals  including  performance  measures  that  would  be  used  to  measure  progress 
toward  attainment  of  these  goals.  The  results  are  to  be  reported  annually  to  Congress  and 
are  subject  to  independent  audits. 

The  Clinger  Cohen  Act  (CCA)  specifically  tasks  agencies  to  develop  a  process  to 
improve  their  planning  for  IT  projects  to  ensure  the  federal  government  gets  the  best 
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possible  return  for  its  investment.  The  Office  of  Management  and  Budgeting  (0MB)  and 
the  Chief  Information  Officer  (CIO)  from  each  executive  department  have  issued 
guidance  for  this  planning  and  are  continually  trying  to  improve  their  investment  model. 
Numerous  studies  of  both  government  and  private  sector  IT  investment  planning  have 
been  conducted  by  the  General  Accounting  Office  (GAO),  including  studies  to  identify 
the  best  practices  in  IT  management  processes.  As  the  watchdog  of  Congress,  GAO  acts 
on  the  request  of  a  congressional  committee  or  member.  Ref  3  was  GAO’s  first 
investigation  into  federal  IT  investment  management  and  issued  as  a  response  to  a 
request  by  then  Senators  Clinger  and  Cohen.  This  report  directly  led  to  passage  of  the 
Information  Technology  Management  Reform  Act. 

B.  PURPOSE 

The  purpose  of  this  thesis  is  twofold.  First,  to  develop  a  model  to  evaluate  a 
government  agency’s  IT  management  processes  with  regards  to  their  compliance  with  the 
provisions  of  Clinger-Cohen  Act  and  to  note  differences  between  the  processes  in-place 
and  those  of  the  best  practices  identified  by  GAO.  Second,  to  use  the  model  developed  in 
this  thesis  to  evaluate  the  IT  management  processes  of  a  specific  federal  agency  to 
determine  if  they  comply  with  the  Clinger-Cohen  Act  and  make  recommendations  to 
improve  their  processes  where  applicable. 

C.  ASSUMPTIONS  AND  LIMITATIONS 

Since  the  Clinger-Cohen  Act  was  implemented  as  an  extension  of  the 

Government  Performance  and  Results  Act,  the  model  developed  in  this  thesis  will 

assume  that  a  strategic  plan  is  fully  developed  and  in  place  for  the  organizations  in 

question.  The  absence  of  such  a  plan  will  not  prevent  the  assessment  of  an  agency’s  IT 

management  processes,  but  the  evaluation  of  the  agency’s  current  processes  will  be 
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limited  to  broad  comparisons  to  the  model  developed  by  this  research.  A  strategic  plan  is 
necessary  to  focus  the  management  policies  and  reap  the  maximum  benefits  envisioned 
by  Clinger-Cohen,  but  neither  the  agency’s  strategic  plan,  nor  their  strategic  planning 
process  will  be  evaluated  in  this  research. 

The  model  developed  in  this  thesis  will  be  broad  enough  to  be  used  as  a  basis  for 
the  evaluation  of  any  government  agency’s  IT  investment  processes.  0MB  and  GAO 
issue  broad  guidance  for  government  agencies,  therefore  research  into  the  basis  of  the 
model  will  be  limited  to  GAO  reports  and  0MB  guidance.  This  thesis  will  use  academic 
sources  to  expand  some  areas  to  support  the  audits  by  GAO  or  0MB  directives,  but  GAO 
guidance  will  form  the  foundation  of  the  model. 

D.  RESEARCH  METHODOLOGY 

1.  Literature  Review 

The  methodology  will  include  a  literature  review  of  the  requirements  of  the 
Clinger-Cohen  Act,  associated  legislation,  and  publicly  available  reports  addressing  the 
best  practices  of  public  and  private  sector  organizations  as  noted  by  GAO.  Additionally, 
a  review  of  research  journals  and  similar  publications  in  the  areas  of  IT  management  and 
strategic  management  will  be  done  to  expand  the  GAO  models. 

2.  Data  Collection 

In  order  to  complete  the  evaluation  of  a  specific  agency’s  IT  management,  a 
review  of  their  existing  processes  will  be  completed  by  studying  relevant  regulations  and 
documented  procedures  and  interviewing  personnel  involved  in  the  management, 
selection,  control,  operation,  or  maintenance  of  IT  investments. 
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E.  BENEFITS  OF  THIS  STUDY 

This  thesis  will  provide  government  managers  and  IT  professionals  with  an 
insight  into  the  background  of  current  initiatives  in  IT  management  reform  and  provide  a 
model  to  help  assess  their  agency’s  IT  management  processes  with  reference  to  the  best 
practices  identified  by  GAO.  The  review  of  a  specific  agency  provides  an  additional 
view  of  managing  IT  and  an  example  of  how  each  agency  needs  to  evaluate  the 
environment  they  work  in  and  choose  the  techniques  that  best  fit  that  environment. 

Additionally  this  thesis  provides  the  selected  agency  with  an  independent, 
objective  assessment  of  the  processes  they  currently  use  to  manage  their  IT  investments 
and  provides  recommendations  for  future  improvements. 

F.  THESIS  ORGANIZATION 

This  Chapter  provides  a  background  into  the  legislation  changing  governmental 
IT  management,  the  purpose  of  this  thesis,  the  assumptions  and  limitations  of  the 
research,  the  methodology  of  the  research,  the  organization  of  the  thesis,  and  the  benefits 
of  this  study. 

Chapter  II  will  address  the  management  of  IT  projects  as  investments  and  outline 
the  phases  of  the  investment  cycle  presented  by  GOA  in  Ref  [3].  The  concept  of  using 
performance  measures  as  a  technique  to  select,  control,  and  evaluate  IT  investments  will 
also  be  introduced. 

Chapter  III  will  develop  a  model  to  compare  an  agency’s  IT  management 
processes  by  adapting  the  critical  processes  identified  with  a  mature,  effective,  and 
efficient  IT  management  plans  with  the  investment  cycle  described  in  Chapter  II. 
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Chapter  IV  will  use  the  model  in  Chapter  III  to  evaluate  the  compliance  with 
CCA  and  the  effectiveness  of  IT  investment  management  for  the  White  House  Situation 
Support  Staff,  a  small  federal  organization. 

Chapter  V  will  include  conclusions  regarding  IT  investment  management 
processes  and  recommendations  for  further  study,  both  at  WHSSS  and  in  the  general  area 
of  IT  investment  management. 
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II.  MANAGING  IT  PROJECTS  AS  INVESTMENTS 

Too  often,  IT  projects  are  viewed  as  expenses  and  no  analysis  is  performed  to 

measure  the  expected  benefits  for  the  dollars  expended.  Additionally,  accoimting  for  an 
IT  project’s  total  costs  is  made  difficult  by  the  practice  of  using  procurement  and 
operational  funds  simultaneously  or  not  fully  associating  all  costs  with  the  project.  In 
order  to  ftxlly  disclose  the  funds  spent  on  IT  projects  and  choose  the  projects  that  provide 
the  best  return  in  benefits  realized  for  the  funds  spent,  a  technique  to  better  manage  IT 
projects  needs  to  be  developed. 

A.  THE  IT  MANAGEMENT  PROCESS 

Since  the  enactment  of  CCA  in  1996,  several  studies  have  been  completed  to 
determine  the  most  efficient  management  processes  for  IT  projects  as  well  as  attempt  to 
develop  a  model  that  can  be  applied  imiformly  and  government- wide  for  ease  of  review 
and  approval  of  IT  Projects.  To  date,  the  focus  of  executive  departments  in  trying  to 
comply  with  CCA  has  been  in  the  acquisition  of  large  (Department  wide)  IT  projects  and 
in  trying  to  establish  a  Departmental  model  for  each  smaller  agency  to  apply  to  its  own 
individual  projects.  There  has  been  little  progress  in  addressing  the  questions  facing  the 
smaller  agencies.  Specifically,  how  to  choose  the  investment  with  the  best  return  and 
how  to  manage  the  required  processes  with  a  limited  number  of  personnel? 

Another  difficult  hurdle  is  transforming  the  agency’s  current  IT  management 
technique  for  different  projects  in  various  stages  of  development  into  the  investment 
cycle  framework  developed  by  GAO  in  Ref  [3].  After  enactment  of  CCA,  GAO 
developed  the  investment  cycle  as  a  model  to  assist  federal  agencies  in  starting  to  design 
IT  investment  management  processes.  GAO  has  addressed  the  transformation  problem 

with  the  Information  Technology  Investment  Management  (ITIM)  framework  in  Ref  [1]. 
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The  ITIM  framework  is  modeled  after  the  Software  Engineering  Institute’s  (SEI) 
fi-amework  for  assessing  an  organization’s  software  management  ability  [Ref  l:p.l39]. 
The  ITIM  fi-amework  uses  “levels  of  maturity”  to  identify  management  needs.  GAO 
[Ref.  1]  not  only  defines  the  maturity  levels  of  the  fi-amework  and  their  critical  aspects, 
but  also  describes  how  an  agency  best  transitions  from  one  maturity  level  to  the  next. 
This  report  is  useful  for  understanding  an  agency’s  IT  management  process  (moving 
between  levels  of  maturity)  and  will  be  addressed  in  Chapter  III  after  the  groundwork  for 
an  IT  management  process  is  discussed. 

This  Chapter  will  present  the  common  factors  of  GAO  reports  [Ref  1  and  2]  to 
describe  a  management  process  for  the  control  of  IT  investments.  This  should  be 
applicable  to  all  IT  investments  regardless  of  agency  size  or  complexity  of  the  IT  project. 
These  factors  include:  the  organizational  attributes  critical  to  successful  IT  investments, 
the  phases  of  the  IT  investment  cycle,  and  IT  performance  measures. 

B.  ATTRIBUTES  CRITICAL  TO  SUCESSFUL  IT  INVESTMENTS 

While  each  phase  of  the  management  process  contains  unique  requirements  to 
ensure  the  success  of  the  IT  investment,  an  organization  must  master  three  distinct 
attnbutes  for  successful  implementation.  GAO  [Ref.  3]  identifies  these  attributes  as: 
senior  management  involvement,  mission  focus  of  projects,  and  the  portfolio  approach  to 
IT  investments. 

1.  Senior  Management  Involvement 

Managers  with  the  authority  to  purchase,  modify,  and  cancel  projects  must  be 
fully  integrated  into  the  IT  investment  process.  Usually  these  are  senior  managers.  A 
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communication  process,  both  routine  meetings  for  planning  and  status  reporting,  and  a 
separate  technique  for  reporting  problems,  must  be  formally  implemented. 

The  planning  for  new  investments,  resource  allocation  for  the  overall  IT  portfolio, 
and  the  validation  of  costs  and  benefits  are  made  in  the  budget  formulation  and 
submission  part  of  the  year.  The  communication  process  must  provide  managers,  who 
have  the  authority  to  modify  or  cancel  IT  projects,  with  the  knowledge  of  which  IT 
projects  are  having  problems  that  are  not  being  corrected  at  lower  levels  and  are  in 
trouble. 

Appropriate  review  processes,  documentation  requirements,  selection  criteria,  and 
dollar  thresholds  to  determine  the  level  of  management  required  for  decision-making 
must  be  developed  for  each  type  of  investment  and  stage  of  development.  This  ensures 
each  investment  gets  the  scrutiny  warranted  by  the  amount  of  investment  dollars 
involved.  The  size  of  the  agency  and  therefore  the  depth  of  management  layers  will 
determine  the  complexity  of  implementing  senior  management  involvement. 

2.  Mission  Focus  of  Project 

The  strategic  plan  consists  of  the  organization’s  mission,  goals,  and  objectives. 
Performance  measures  must  be  linked  to  strategic  planning  as  required  by  CCA  and 
GPRA.  This  requires  developing  long-term  strategic  goals,  setting  annual  organizational 
performance  targets  in  support  of  those  objectives,  and  annually  evaluating  performance 
against  those  targets. 

In  order  for  executive  departmental  goals  to  be  met,  each  smaller  agency  within 
an  executive  department  must  ensure  that  their  objectives  support  their  parent  agency’s 

goals.  This  results  in  the  department’s  goals  being  addressed  in  increasing  detail  as  the 
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process  moves  down  through  the  organi2ational  structure.  Likewise,  an  IT  office’s  goals 
need  to  be  expressed  in  terms  describing  how  they  will  specifically  support  their  agency’s 
objectives  instead  of  broad  terms  such  as  “provide  cutting  edge  IT  products.”  Without 
sufficiently  specific  translation  of  goals,  people  will  not  focus  on  their  organization’s  true 
mission. 

IT  projects  must  now  be  selected  to  help  the  agency  meet  their  objectives.  If  an 
IT  investment  cannot  be  proven  to  measurably  improve  agency  mission  performance,  that 
investment  should  not  be  made.  Mission  benefit,  not  cost  or  schedule  constraints,  must 
be  the  main  measure  of  success  of  an  IT  investment.  This  point  must  be  stressed  in  the 
selection  of  IT  investments  and  will  help  determine  if  any  corrective  actions  are  needed 
while  the  IT  investment  is  being  developed  and  deployed. 

The  development  of  the  strategic  plan  and  the  performance  targets  is  instrumental 
in  the  implementation  of  IT  investment  management  since  the  investments  will  be  chosen 
to  help  the  agency  achieve  its  objectives  and  mission.  The  performance  measures  that  IT 
investments  are  evaluated  and  ranked  by  must  be  focused  on  the  performance  measures 
used  by  the  agency’s  senior  managers  to  measure  progress  towards  these  targets.  This  is 
also  related  to  the  involvement  of  senior  management,  as  the  success  of  any  strategic  plan 
is  linked  to  both  the  involvement  and  commitment  of  the  agency  leadership. 

3.  The  Portfolio  Approach  to  IT  Investments 

Most  organizations  have  numerous  IT  projects  in  different  stages  of  maturity, 
ranging  fi-om  new  projects  to  those  in  the  final  stages  of  completion.  In  order  to  properly 
control  all  IT  investments,  the  agency  must  define  a  portfolio  consisting  of  all 
investments  in  every  stage  of  development  and  of  every  type.  The  stages  of  development 
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include  concept  exploration  through  operational  use,  and  must  not  be  confused  with  the 
phases  of  the  IT  investment  cycle  (which  are  explained  in  the  next  section  of  this 
chapter),  although  they  are  related.  Investments  that  are  purely  research  and  development 
should  be  included  in  the  agency’s  R&D  portfolio,  rather  than  in  its  IT  investment 
portfolio.  Also,  different  types  of  IT  investments  should  be  identified  such  as  mission 
critical,  cross-functional,  or  administrative.  “Mission  critical”  IT  investments  help  the 
agency  accomplish  an  agency  mission  (as  identified  in  the  agency’s  strategic  plan). 
“Cross-functional”  IT  investments  address  one  or  more  missions  and/or  one  or  more 
administrative  functions,  while  “administrative”  IT  investments  address  only  agency 
administration. 

By  building  an  all-inclusive  portfolio  of  investments,  resources  can  be  allocated  to 
investments  after  they  are  ranked  based  on  their  costs  and  benefits. 

C  PHASES  OF  THE  IT  INVESTMENT  CYCLE 

1.  Introduction 

The  intrinsic  part  of  the  IT  management  process  is  the  description  of  the  phases  of 

the  investment  cycle  as  reconunended  by  GAO  [Ref  l:p.  3].  Figure  1  displays  the  three 

phases  of  the  cycle:  Selection,  Control,  and  Evaluate.  The  phases  are  connected  by  both 

process  and  information  flow.  Data  collected  from  investments  in  the  Control  or 

Evaluate  phases  provides  feedback,  which  can  improve  decisions  made  in  future 

Selection  phase  meetings.  Likewise,  decisions  made  in  the  Selection  phase  to  change  the 

agency’s  portfolio  will  affect  what  options  are  available  in  the  Control  phase.  While  a 

single  IT  investment  will  proceed  through  the  cycle  in  order  from  Selection  to  Control  to 

Evaluate,  the  agency’s  IT  portfolio  most  likely  contains  investments  in  each  phase.  Thus, 

management  is  constantly  documenting  data  collected  and  decisions  made  in  all  phases 
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throughout  the  business  cycle.  Each  of  the  three  phases  will  be  discussed  in  the 
following  sections. 


Figure  1.  Phases  of  the  IT  Investment  Cycle  After  Ref  [1] 


2.  Phase  One:  Selection 

The  purpose  of  the  Selection  phase  is  to  build  the  agency’s  IT  portfolio.  This  is 
accomplished  through  three  processes:  Screening,  Ranking,  and  Allocation.  The 
Selection  phase,  unlike  the  Control  and  Evaluate  phases,  is  not  ongoing,  but  normally 
occurs  at  a  distinct  point  in  the  agency’s  operating  cycle.  The  Selection  phase  may  occur 
annually,  quarterly,  or  at  any  time  during  the  year.  Government  agencies  may  find  it 
most  useful  to  conduct  the  selection  phase  annually  in  conjunction  with  their  budget 
submission  process. 
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fl.  The  Screening  Process 

The  GAO  describes  the  screening  process  as  the  comparison  of  all 
investments  (potential  as  well  as  current  investments)  against  a  imiform  set  of  established 
screening  criteria  and  thresholds  in  order  to  determine  which  projects  meet  the  minimum 
requirements  for  continuing  review  and  at  what  organizational  level  the  investments  must 
be  reviewed.  The  policies  of  the  agency  will  delineate  the  thresholds  and  the 
organizational  level  of  review  required  for  different  sized  projects.  Examples  of 
thresholds  that  would  have  to  be  met  include  a  minimum  return  on  investment  (ROI),  or  a 
maximum  cost  amount  based  on  a  percentage  of  the  normal  annual  IT  budget. 

b.  The  Ranking  Process 

The  ranking  process  will  assign  priorities  to  the  IT  investment  projects 
that  survive  the  screening  process  using  both  measures  from  the  screening  process  and 
any  additional  measures  needed  to  breakout  each  potential  investment’s  rank  or  prioritize 
the  projects  from  highest  to  lowest.  In  order  to  accomplish  the  ranking,  several  measures 
are  necessary.  At  a  minimum,  measures  of  cost  vs.  benefit  (i.e.  ROI),  risk  to  the  agency, 
and  mission  accomplishment  (i.e.  strategic,  tactical,  or  administrative)  must  be  used.  The 
measures  can  then  be  weighted  based  on  the  agency’s  abilities  and  mission.  For  instance, 
projects  addressing  a  mission  that  is  currently  not  being  met  must  rank  above  projects 
aimed  at  improving  other  missions  even  when  the  ROI  is  lower  or  higher  risks  may  be 
involved.  Comparisons  of  the  weighted  scores  for  each  investment  can  then  be  used  to 
rank  the  investments. 

c.  The  Allocation  Process 

The  last  process  in  the  Selection  phase  is  to  allocate  funds  for  the  next 

business  cycle  to  investments  based  on  the  priorities  established  in  the  ranking  process. 
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Possible  outcomes  in  the  allocation  are  continuing  the  funding  level  of  an  investment, 
increasing  the  funding,  lowering  the  funding  level,  cancellation  of  an  existing  investment, 
or  deferment  of  a  potential  investment  due  to  lack  of  resources.  This  final  process  is 
necessary  to  allocate  the  limited  funds  to  the  projects  that  provide  the  most  benefit  to  the 
agency  in  the  form  of  mission  accomplishment,  ROI,  and  at  manageable  risk  levels.  If 
the  IT  funds  were  unlimited,  all  projects  could  be  funded  making  the  ranking  and 
allocating  processes  unnecessary. 

This  whole  set  of  processes  in  the  Selection  phase  is  summarized  by  the 
GAO  as  follows: 

The  Selection  phase  helps  ensure  that  the  organization  (1)  selects  those 
IT  projects  that  will  best  support  mission  needs  and  (2)  identifies  and 
analyzes  a  project's  risks  and  proposed  benefits  before  a  significant 
amount  of  project  funds  are  spent.  A  critical  aspect  of  this  phase  is 
management  understanding  and  participation  and  decision-making  that  is 
driven  by  accurate,  up-to-date  data  and  an  emphasis  on  using  IT  to 
enhance  mission  performance.  [Ref  2:p.7] 

3.  Phase  Two:  Control 

The  purpose  of  the  Control  phase  is  to  track  the  investment  through  its 
development  and  deployment  to  ensure  the  appropriate  level  of  performance  is  being 
gained  from  each  investment.  This  helps  prevent  the  loss  of  resources  on  investments 
that  are  not  helping  the  agency  improve  its  strategic  mission  accomplishment. 

Once  investments  are  selected  and  funded,  they  are  continuously  and  consistently 
managed  and  controlled.  The  progress  of  the  investment  is  measured  against  projected  or 
expected  cost,  schedule,  and  mission  benefits.  Performance  reviews  of  investments  are 
conducted  at  key  points  in  the  investment’s  development.  The  periodicity  of  these 
reviews  will  be  set  at  the  beginning  of  the  IT  project’s  development  based  on  its  cost. 
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complexity,  or  risk  and  may  be  altered  based  on  performance  during  development.  High 
cost,  complexity,  or  risk  projects  require  more  frequent  control.  The  periodicity  of  these 
reviews  can  also  be  changed  as  necessary  depending  on  the  size,  risk,  and  performance  of 
the  project  during  its  development  and  deployment. 

Performance  measures  used  to  track  the  investment  will  have  different  thresholds. 
Each  investment  will  have  a  predicted  value  (i.e.  the  “good”  or  “green”  value)  of  the 
final  project  for  each  performance  measure.  Investments  in  development  or  deployment 
with  all  performance  measures  in  the  “green”  are  progressing  as  expected.  Project 
managers  will  set  warning  thresholds  (i.e.  the  “warning”  or  “yellow”  value)  at  which  the 
project  manager  decides  on  the  corrective  action  that  is  required  and  documents  his  or  her 
decision  and  any  action  taken.  When  the  danger  threshold  (i.e.  the  “danger”  or  “red” 
value)  is  reached,  management  involvement  is  increased  by  trouble  reports  and  the  senior 
managers  will  have  to  decide  whether  or  not  the  project  needs  further  corrective  action, 
modification,  or  cancellation  based  on  investment  performance  and  response  to  corrective 
action  taken  at  lower  levels. 

The  setting  of  each  of  these  thresholds  is  very  critical  to  the  management  process. 
McGee  and  Prusak,  members  of  the  Ernst  &  Young  consulting  firm,  stress  that 
“orgamzations  must  treat  metrics  as  attention-focusing  devices  instead  of  scorecards” 
[Ref.  4:p.l92].  This  must  mean  that  the  “yellow”  values  show  situations  that  require 
attention,  and  administrators  should  not  be  too  quick  to  terminate  or  even  modify 
projects.  Likewise,  project  managers  need  to  imderstand  that,  even  though  the 
investment  survived  the  selection  process,  a  point  may  come  where  the  returns  delivered 
by  an  investment  when  weighed  against  the  resources  allocated  to  that  project  may 
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warrant  modifications  or  even  termination.  This  emphasizes  the  importance  of 
documenting  corrective  actions  taken  or  the  basis  for  not  taking  action. 

4.  Phase  Three:  Evaluate 

The  purpose  of  the  Evaluate  phase  is  to  examine  the  lessons  learned  fi'om  each 
specific  project  and  use  these  to  adjust  the  processes  in  the  Selection  and  Control  phases 
to  continually  improve  the  agency’s  management  of  its  IT  portfolio. 

After  a  project  has  reached  its  final  endpoint  (usually  full  implementation  or 
cancellation),  a  final  review  is  completed.  GAO,  in  its  “Guide  for  Evaluating  Agencies’ 
IT  Investment  Decision-making”,  describes  three  major  areas  that  the  final  review  must 
examine:  effect  on  customers,  impact  on  agency  mission,  and  technical  capability. 

The  effect  on  customers”  must  be  examined  to  determine  if  the  recipients  of  the 
IT  services  or  users  of  the  IT  technology  realized  the  desired  benefits.  In  addition  to  the 
plaimed  benefits,  the  review  must  determine  the  impact  of  any  unplanned  aspects  of  the 
project.  This  includes  negative  aspects  of  the  project  such  as  added  training  or 
inefficiency  due  to  increased  project  complexity  as  well  as  any  unplanned  benefits  that 
may  have  been  realized.  This  will  usually  require  interviewing  or  surveying  the  project’s 
users  and  maintainers. 

The  impact  on  agency  mission”  must  likewise  be  examined  for  both  unplanned 
effects  on  mission  as  well  as  whether  or  not  the  intended  affect  on  agency  mission  was 
realized. 

The  “technical  capability”  of  the  project  can  be  assessed  by  the  trends  displayed 
by  the  history  of  performance  measures  during  the  life  of  the  project  (maintained  during 

the  Control  phase)  compared  with  the  desired  goals  expressed  in  the  Selection  phase. 
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Other  aspects  of  the  technical  capability  of  the  project  include  employee  satisfaction  with 
the  system,  the  extent  to  which  the  system  was  utilized  by  employees  and  customers 
alike,  and  the  expertise  displayed  by  the  technicians  responsible  for  the  project. 

The  most  difficult  aspect  of  the  final  review  is  to  find  benefits  and/or 
shortcomings  that  were  not  necessarily  measured  by  the  performance  measures  tracked  in 
the  Control  phase.  Since  the  goal  of  the  Evaluate  phase  is  to  improve  the  processes  in  the 
Selection  and  Control  phases,  the  agency  must  strive  to  find  which  positive  or  negative 
attributes  were  realized  without  being  explicitly  monitored.  Agency  managers  can  then 
modify  the  Selection  and  Control  of  investments  to  minimize  the  negative  attributes  and 
capitalize  on  the  positive. 

Last,  the  final  review  procedures  must  be  formalized  in  agency  policy.  A 
consistent,  thorough  procedure  must  be  utilized  for  all  projects.  The  agency  must  also  be 
able  to  identify  shortcomings  in  its  evaluations  and  improve  on  these  procedures  should 
they  find  better  techniques  or  documentation  to  do  so.  The  policy  must  formalize  the 
procedure  to  be  used  so  that  it  can  be  clearly  communicated  and  consistently  applied. 
However,  policy  must  be  flexible  enough  to  allow  continuous  improvements  to  the 
Evaluate  phase  by  allowing  application  of  lessons  learned  from  each  experience. 

D.  IT  PERFORMANCE  MEASUREMENTS 

Once  the  agency  understands  the  IT  investment  cycle  and  the  basic  processes 
required  to  manage  the  cycle,  a  method  must  be  developed  to  measure  the  qualities 
(positive  and  negative)  of  IT  investments.  These  measurements  are  used  to  rank 
investments  against  each  other  in  the  Selection  phase  and  are  used  to  track  an 
investment’s  progress  in  the  Control  phase.  A  wise  selection  of  “performance 
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measurements”  to  manage  IT  investments  is  crucial  to  the  success  of  an  agency’s  IT 
management  processes. 

1.  Introduction 

As  discussed  in  the  Selection  and  Control  phases,  consistent  and  reliable  metrics 
must  be  used  when  comparing  investments  or  tracking  their  performance  through 
development.  There  are  countless  performance  measures  available  for  the  agency  to  use, 
but  they  must  be  chosen  to  address  three  features  sought  by  GAO  auditors:  repeatability, 
efficiency,  and  completeness.  As  seen  in  Figure  2  this  adds  a  measure  of  breadth  to  the 
Select,  Control,  and  Evaluate  phases. 

They  examine  each  depth  of  each  phase  for  the  critical  features  of  its  performance 
measures,  namely  repeatability,  efficiency,  and  completeness.  A  process  is  “repeatable” 
when  the  documentation  that  describes  the  data  required  and  technique  used  to  make 
decisions  allows  different  personnel  to  come  to  the  same  decision  when  the  process  is 
accomplished  in  another  investment  cycle.  “Efficiency”  in  a  process  is  achieved  when  all 
participants  in  the  process  fully  understand  what  is  expected  of  the  process  and  both 
wasted  time  and  resources  are  eliminated  from  the  process.  “Completeness”  is  a  direct 
result  of  the  process  documentation  conveying  the  required  inputs  and  outputs  of  the 
process  such  that  no  data  or  analyses  are  missed. 
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Figure  2.  Three  Dimensions  of  the  IT  Investment  Evaluation  Approach  From  Ref.  [2] 

In  addition,  as  shown  in  Figure  2,  GAO  expands  the  IT  management  process  by 
adding  a  third  dimension  consisting  of:  process,  data,  and  decisions. 

The  critical  aspect  of  choosing  the  appropriate  performance  measures  is 
connecting  the  investment  to  the  agency  mission  it  supports,  and  determining  what  output 
will  be  measured  to  track  how  well  the  desired  outcome  is  being  achieved.  The 
importance  of  identifying  the  desired  outcomes  and  relating  them  to  measurable  outputs 
cannot  be  stressed  enough.  The  best  laid-out  strategy  will  never  be  realized  if  the  desired 
outcomes  are  not  tied  to  appropriate  outputs  that  can  be  measured.  These  measurements 
are  the  indicators  used  to  provide  feedback  for  corrective  actions  in  the  system  to 
continually  strive  towards  the  desired  outcomes.  Chapter  3  will  discuss  the  use  of  the 
“balanced  scorecard”  to  tie  these  missions,  projects,  outputs,  and  outcomes  together. 
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The  remainder  of  this  chapter  will  discuss  two  broad  areas  that  must  be  measured 
to  track  IT  project  performance  and  finally  how  these  measurements  can  be  compared 
and  used  to  rank  projects.  GSA’s  “Performance  Based  Management:  Eight  Steps  to 
Develop  and  Use  Information  Technology  Performance  Measures  Effectively”  discusses 
the  measurements  of  effectiveness,  efficiency,  and  the  use  of  information  economics  to 
apply  these  measurements  to  a  scorecard. 

2.  Effectiveness 

Effectiveness  is  the  measure  of  how  well  an  objective  is  being  met.  The  agency 
must  carefully  choose  the  outputs  measured  and  avoid  choosing  inputs.  An  example  of 
this  mistake  is  an  office  that  tracks  the  number  of  requests  filled.  The  number  of  requests 
filled  is  tied  directly  to  the  number  submitted,  which  is  an  input  into  the  system  not  an 
output.  A  better  choice  in  this  case  would  be  the  percentage  of  reports  processed  within 
two  working  days.  The  percentage  removes  the  input  (although  it  is  still  related  to  the 
number  of  inputs)  and  instead  emphasizes  the  attainment  of  a  business  goal,  that  all 
reports  be  processed  within  two  working  days. 

The  issue  of  effectiveness  must  be  addressed  in  order  to  enhance  the  completeness 
of  the  agency’s  performance  measurements.  This  relates  back  to  the  point  that  no  project, 
no  matter  how  well  it  performs  with  respect  to  schedule  and  cost,  should  be  advanced  if  it 
does  not  help  the  agency  achieve  its  missions. 

3,  Efficiency 

Efficiency  is  the  measure  of  how  well  a  project  is  providing  an  output  comparing 
benefits  to  the  cost  of  its  operation.  The  most  recognized  measurement  in  this  area  is 
return  on  investment  (ROI).  ROI  is  the  benefits  to  the  agency  (measured  in  dollars) 
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provided  by  the  investment  divided  by  cost  of  acquiring  and  operating  an  investment. 
This  calculation  is  difficult  with  IT  projects  because  there  is  usually  a  myriad  of 
intangible  benefits  that  IT  investments  both  deliver  and  promise  to  deliver.  IT 
corporations  have  attempted  to  improve  the  ROI  of  IT  investments  by  introducing  the 
concept  of  “enhanced  ROI.”  The  enhanced  ROI  is  no  different  from  the  conventional 
ROI  calculation  except  the  intangible  benefits  are  given  a  dollar  value. 

The  most  difficult  aspect  of  the  efficiency  measurement  is  the  inclusion  of 
intangibles.  This  has  to  be  approached  from  two  perspectives.  First,  the  intangibles  must 
be  addressed.  This  may  mean  starting  from  a  baseline,  such  as  using  the  IT  system 
already  in  place.  Then  the  intangibles  can  be  measured  as  incremental  changes  fi-om  the 
baseline.  Second,  the  measurement  of  intangibles  needs  to  be  formalized  such  that  the 
process  is  consistently  applied  across  all  investments  in  the  portfolio,  not  just  projects 
competing  for  the  same  mission,  since  all  investments  “compete”  against  each  other 
when  they  are  ranked  in  the  Selection  phase. 

4.  Information  Economics 

In  order  to  formalize  the  measurement  of  intangibles  when  determining  the  value 
of  IT  investments,  the  General  Services  Administration  recommends  the  use  of  score 
factors,  as  developed  by  Parker  and  Benson  [Ref  5].  These  factors  help  to  improve  the 
ROI  calculations  by  providing  factors  to  assess  the  value  and  risks  of  IT  projects.  These 
factors  can  then  be  used  to  build  a  scorecard  for  each  IT  investment  that  will  be  used  to 
rank  all  IT  investments  in  the  portfolio.  Agencies  can  use  all  or  some  of  the  factors,  or 
develop  their  own. 

The  Information  Economics  Scorecard  factors  [Ref  5:p.53-55]  are: 
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Enhanced  Return  on  Investment  (ROI) —  Assesses  the  cost-benefit 
analysis  plus  the  benefits  created  by  the  IT  investment  on  other  parts  of 
the  organization.  Parker  and  Benson  provide  techniques  for  assessing  costs 
and  benefits  of  the  impact  of  an  IT  investment  on  other  departments  of  the 
organization.  They  also  describe  techniques  for  quantifying  the  benefits 
associated  with  increasing  the  value  of  a  function.  For  example,  electronic 
form  processing  provides  a  data  entry  clerk  with  the  capability  to  process 
claims,  a  higher  value  function. 


Strategic  Match  (SM)  —  Assesses  the  degree  to  which  the  proposed 
project  corresponds  to  established  agency  strategic  goals.  This  factor 
emphasizes  the  close  relationship  between  IT  planning  and  corporate 
planning  and  measures  the  degree  to  which  a  potential  project  contributes 
to  the  strategy.  Projects  that  are  an  integral  and  essential  part  of  the 
corporate  strategy  receive  a  higher  score  than  those  that  are  not.  Strategic 
Match  assesses  the  extent  to  which  an  IT  investment  enables  movement 
towards  long-term  direction. 


Competitive  Advantage  (CA)  —  Assesses  the  degree  to  which  projects 
create  new  business  opportunities,  facilitate  business  transformation  (e.g., 
inter-orgamzation  collaboration  through  electronic  commerce),  increase 
the  agency’s  competitiveness  or  improve  the  agency’s  reputation  or  image. 
Competitive  Advantage  requires  placing  a  value  on  a  project’s 
contribution  toward  achieving  one  or  more  of  these  objectives. 

Management  Information  (MI)  —  Assesses  a  project’s  contribution  to 
nianagement’s  need  for  information  about  core  activities  that  involve  the 
direct  realization  of  the  mission,  versus  support  activities.  Measuring  a 
project’s  conribution  to  the  core  activities  of  the  business  implies  that  the 
agency  has  identified  its  critical  success  factors.  This  measurement  is 
obviously  subjective  because  improved  management  information  is 
intangible,  but  the  benefit  measurement  can  be  improved  if  the  agency 
first  defines  those  core  activities  critical  to  its  success,  and  then  selects  a 
general  strategy  to  address  these  issues. 

Le^slatiye  Implementation  (LI)  —  Assesses  the  degree  to  which  the 
project  implements  legislation.  Executive  Orders  and  regulatory 
requirements.  For  example.  Federal  law  requires  INS  to  process 
passengers  arriving  at  airports  from  international  flights  within  45 
minutes.  A  project  receives  a  high  score  if  it  directly  implements 
legislation;  a  moderate  score  if  it  indirectly  implements  legislation;  and  no 
score  if  the  project  does  neither. 


Organizational  Risk  (OR)  —  Assesses  the  degree  to  which  an 
information  systems  project  depends  on  new  or  untested  corporate  skill, 
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management  capabilities  and  experience.  Although  a  project  may  look 
attractive  on  other  dimensions  and  the  technical  skills  may  be  available, 
unacceptable  risks  can  exist  if  other  required  skills  are  missing.  This  does 
not  include  the  technical  organization,  which  will  be  measured  on  another 
dimension.  Organizational  risk  also  focuses  on  the  extent  to  which  the 
organization  is  capable  of  carrying  out  the  changes  required  by  the  project, 
that  is,  the  user  and  business  requirements.  For  example,  a  high  score 
reflects  that  the  business  domain  organization  has  no  plan  for 
implementing  the  proposed  system;  management  is  uncertain  about 
responsibility;  and  processes  and  procedures  have  not  been  documented. 

Strategic  IS  Architecture  (SA)  — ^Assesses  the  degree  to  which  the 
proposed  project  fits  into  the  overall  information  systems  direction  and 
conforms  to  open-system  standards.  It  assumes  the  existence  of  a  long¬ 
term  information  systems  plan  —  an  architecture  or  blueprint  that  provides 
the  top-down  structure  into  which  future  data  and  systems  must  fit. 

Definitional  Uncertainty  (DU)  —  A  negatively  weighted  factor  that 
assesses  the  degree  of  specificity  of  the  user’s  objectives  as  communicated 
to  the  information  systems  project  personnel.  Large  and  complex  projects 
that  entail  extensive  software  development  or  require  many  years  to 
deliver  have  higher  risks  compared  to  those  projects  segmented  into 
modules  with  near-term  objectives. 

Technical  Uncertainty  (TU)  —  Assesses  a  project’s  dependence  on  new 
or  untried  technologies.  It  may  involve  one  or  a  combination  of  several 
new  technical  skill  sets,  hardware  or  software  tools.  The  introduction  of  an 
untried  technology  makes  a  project  inherently  risky. 

IS  Infrastructure  Risk  (IR)  — ^Assesses  the  degree  to  which  the  entire  IS 
organization  is  both  required  to  support  the  project,  and  prepared  to  do  so. 

It  assesses  the  environment,  involving  such  factors  as  data  administration, 
communications  and  distributed  systems.  A  project  that  requires  the 
support  of  many  functional  areas  is  inherently  more  complex  and  difficult 
to  supervise;  success  may  depend  on  factors  outside  the  direct  control  of 
the  project  manager 

Once  a  numerical  score  is  generated  for  each  category  based  on  a  scale  defined  by 
the  agency’s  policy,  they  can  be  used  to  build  a  scorecard  by  summing  the  common 
factors  together  in  common  groups  (financial,  benefits,  and  risk).  Scores  for  financial 
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return,  expected  benefits,  and  risk  can  then  be  compared  for  each  IT  investment  to  assist 
the  ranking  in  the  Selection  process. 

An  integrated  product  team  (IPT)  formed  by  the  Chief  Information  Officer  of  the 
Navy  developed  a  simpler  information  economics  scorecard  addressing  only  financial 
costs  and  gains  to  calculate  a  ROI,  risks  involved,  and  mission  accomplishment  [Ref. 
6:ppJ-A-6  to  J-A-17].  Whatever  scorecard  approach  is  used,  the  complexity  of  the 
analyses  used  needs  to  fit  the  size  and  complexity  of  both  the  agency  and  the  investments 
undertaken. 

Finally,  the  measurements  made  by  information  economics  scorecards  must  be 
continually  examined  in  all  three  phases  (Selection,  Control,  Evaluate)  to  determine  if 
they  are  useful,  and  if  so,  how  they  can  be  improved.  Since  the  scorecard  is  made  to 
assist  in  the  ranking  of  projects  in  the  Selection  phase  and  tracking  of  investment 
progress  in  the  Control  phase,  the  measurements  made  by  information  economics  must  be 
continually  evaluated  to  ensure  they  are  generating  the  scores  necessary  to  help  the 
agency  make  sound  decisions. 
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III.  IMPLEMENTING  INFORMATION  TECHNOLOGY 
INVESTMENT  MANAGEMENT  IN  SMALL  AGENCIES 


A.  THE  ITIM  MATURITY  STAGES 
1.  Overview 

The  previous  chapter  outlined  GAO’s  three-phased  framework  of  Select,  Control, 
and  Evaluate  in  the  IT  investment  cycle.  In  “Information  Technology  Management:  A 
Framework  for  Assessing  and  Improving  Process  Maturity,”  GAO  expanded  the  three- 
phase  framework  for  IT  project  management  previously  promulgated  by  themselves  and 
0MB  by  adapting  the  idea  of  “maturity  stages”[Ref  1].  GAO  decided  on  a  five-stage 
maturity-based  framework  based  on  general  industry  acceptance  of  maturity-based 
frameworks  (such  as  the  Software  Engineering  Institute’s  copyrighted  Capability 
Maturity  Model)  and  GAO’s  working  experience  in  applying  maturity-based  methods  in 
IT  audits  and  evaluations  [Ref.  I:p.l39]. 

Figure  3  illustrates  the  five  stages  of  increasing  maturity  along  with  the  “critical 
processes”  that  are  used  to  characterize  each  of  the  stages.  The  left  side  of  the  figure 
shows  the  wide  range  of  investment  management  from  a  narrow  project-oriented 
approach  to  those  that  encompass  enterprise-wide  strategic  concerns. 

The  framework  in  Figiue  3  can  be  used  to  identify  a  specific  agency’s  IT 
management’s  level  of  ability  in  one  of  the  five  distinct  maturity  stages.  The  framework 
is  cumulative  so  that  an  agency  evaluated  to  operate  at  a  specific  maturity  stage  is 
assumed  to  be  managing  the  critical  processes  at  this  stage  as  well  as  the  prior  stages 
outlined  in  Figure  3. 
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As  noted  in  Figure  3,  agencies  at  Stage  One  have  little  awareness  of  how  their  IT 
investments  are  performing  or  how  they  relate  to  the  agency’s  mission.  The  few 
management  processes  that  are  in  place  are  generally  project  oriented  and  outcomes  are 
widely  variable.  Hence  this  stage  can  be  described  as  “undisciplined”. 


Figure  3.  The  Five  Stages  of  Maturity  Within  ITIM  From  Ref  [1] 

At  Stage  Two  agencies  begin  to  implement  investment  selection  and  control 
techniques  such  as  business  needs  identification  for  IT  projects,  project  oversight,  asset 
tracking,  and  the  development  of  the  IT  oversight  board.  These  are  further  enhanced  in 
Stage  Three  by  focusing  on  a  comprehensive  investment  portfolio  approach  and 
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incorporating  portfolio-based  benefit  and  risk  criteria  linked  to  near-term  business  needs 
of  the  agency. 

Stage  Four  improves  the  investment  process  by  instituting  post-implementation 
reviews  and  feedback  from  these  reviews  to  improve  the  Selection  and  Control  phases  of 
the  portfolio.  Plus,  this  stage  introduces  concern  for  succeeding  information  systems  and 
technologies  that  will  replace  high  cost  or  low  value  added  current  investments. 

At  the  final  stage,  Stage  Five,  an  agency  benchmarks  its  IT  investment  processes 
against  best  practices  of  similar  agencies  and  organizations.  Furthermore,  managers 
attempt  to  leverage  their  IT  investments  to  other  operational  and  strategic  activities  across 
the  entire  enterprise.  In  this  case,  IT  is  the  driver  of  strategic  direction  and  organizational 
change  instead  of  a  reactive  management  tool. 

GAO  uses  this  framework  to  assess  an  agency’s  IT  management  processes  and  as 
a  guideline  for  organizational  improvement.  The  report  outlines  the  purpose, 
prerequisites,  and  the  activities  involved  in  each  of  the  critical  processes  and  then 
describes  how  an  agency  may  transition  from  one  stage  to  the  next. 

This  useful  framework  will  be  further  developed  in  this  thesis  to  examine  the 
critical  processes  necessary  to  address  all  three  phases  of  the  IT  investment  cycle  and, 
from  this,  to  build  a  model  which  can  be  used  to  assess  an  agency’s  IT  management 
processes.  All  of  the  critical  processes  in  Stages  Two,  Three,  Four,  and  Five  are  required 
to  accomplish  this.  The  critical  processes  of  benchmarking  and  IT-driven  strategic 
business  change  in  Stage  Five  bring  the  agency  to  the  point  where  IT  investments  drive 
the  strategic  direction  of  the  organization.  This  chapter  will  expand  this  idea  and 
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describe  how  to  use  the  ITIM  framework  as  an  assessment  tool  for  the  IT  management 
policy  of  a  government  agency. 

2.  Using  the  ITIM  Framework  to  Build  Agency  Policy 
GAO’s  ITIM  framework  lays  the  groimdwork  for  evaluating  IT  management  by 
addressing  each  phase  of  the  investment  cycle  (Selection,  Control,  and  Evaluate)  in  each 
stage  of  maturity  for  an  agency.  Figure  4  shows  the  three  phases  of  the  investment  cycle 
and  the  proposed  critical  processes  that  were  discussed  above  and  included  in  Stages 
Two,  Three,  Four,  and  Five  of  Figure  3. 


Figure  4.  Linking  ITIM  Critical  Processes  to  the  Investment  Phases 

Once  the  critical  processes  are  identified  with  the  relevant  phases  of  the 
investment  cycle,  they  can  be  applied  to  the  Dimensions  of  the  IT  Investment  Evaluation 
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Approach  outlined  in  Chapter  II.  This  comprehensive  three-dimensional  representation 
of  the  agency’s  IT  management  policy  is  depicted  in  Figure  5. 
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Figure  5.  The  Dimensions  of  an  Agency’s  IT  Management  Policy 


The  agency  can  identify  the  processes  in  its  IT  management  policy  and  compare 
them  to  the  critical  processes  in  the  framework  of  Figure  5  and  evaluate  each  for  the 
necessary  attributes  of  repeatability,  efficiency,  and  completeness. 

The  remainder  of  this  chapter  will  describe  the  purpose  of  and  activities  required 

in  each  of  these  critical  processes  for  each  investment  phase. 

B.  THE  SELECTION  PHASE 

1.  IT  Investment  Board  Operation 

The  IT  investment  board  is  the  formal  body  designated  by  agency  policy  to  make 
decisions  concerning  project  selection,  modifications  to  projects,  and  funding  levels.  IT 
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investment  board  membership  and  an  agency-specific  process  guide  (policies  and 
procedures  that  define  board  operations)  are  an  essential  part  of  an  agency’s  IT 
management  policy. 

Both  senior  line  managers  of  major  functions  or  operations  and  IT  personnel  must 
be  represented  in  the  board  membership.  The  size  of  the  board  depends  on  the  size  and 
needs  of  the  specific  organization.  The  members  of  the  board  must  be  trained  to 
understand  their  duties  on  the  board  and  these  duties  must  also  be  outlined  in  agency 
policy  to  ensure  both  repeatability  and  ease  of  review  by  agency  administrators  and 
outside  agencies. 

The  board  s  process  guide  must  identify  specific  events  where  decisions  must 
be  made  by  the  board.  An  equally  important  section  of  this  guide  outlines  the 
environment  in  which  the  agency  operates  in  terms  of  legal  constraints  or  requirements 
and  agreements  with  other  agencies  that  must  be  honored,  and  how  these  affect  the 
different  decisions  that  must  be  made.  This  process  guide  will  be  the  foundation  of  many 
of  the  other  critical  processes  since  it  will  define  the  format  of  deliverables  provided  to 
this  board  for  use  in  decision-making. 

Agency  commitment  to  the  board  must  exist  in  two  important  areas.  The 
agency’s  top  executives  must  provide  strong  leadership  in  defining  the  board’s 
responsibilities  and  membership  and  they  must  support  and  carry  out  board  decisions. 
There  may  be  exceptions  where  top  administrators  act  contrary  to  investment  board 
decisions,  but  these  occurrences  will  be  rare  when  the  policies  and  procedures  of  the 
investment  board  are  properly  aligned  with  agency  missions  and  objectives. 
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2.  Business  Needs  Identification  for  IT  Projects 

In  order  to  meet  the  requirements  of  CCA  and  ensure  all  IT  investments  help  the 
agency  meet  its  mission  goals,  each  investment  must  be  tracked  back  to  the  agency 
mission(s)  that  it  addresses. 

One  way  of  connecting  IT  projects  with  the  agency’s  missions  is  by  use  of  the 
“balanced  scorecard.”  The  balanced  scorecard  can  be  used  to  express  the  vision  and 
missions  of  the  agency  in  terms  of  the  value  seen  by  its  different  stakeholders.  Once  the 
value  seen  by  stakeholders  is  described,  the  IT  investment(s)  that  best  serve  these 
stakeholders  can  be  chosen  and  the  performance  measures  to  be  used  to  control  projects 
can  be  identified. 

Figure  6  outlines  a  tool  that  can  be  used  to  link  the  agency’s  strategy  to  the 
various  perspectives  that  will  perceive  success  or  failure.  The  critical  success  factors 
must  be  determined  from  each  perspective  as  well  as  the  IT  investments  that  address 
these  needs.  Each  perspective  (financial,  customers,  internal  business,  and  innovation 
and  learning)  will  have  different  critical  success  factors  and  projects  that  address  these 
factors.  Performance  measures  are  selected  to  track  the  investments’  progress  towards 
satisfying  the  respective  critical  success  factors. 

Once  these  connections  between  missions,  stakeholders’  needs,  investments,  and 
performance  measures  are  made,  existing  investments  can  be  matched  to  their  respective 
mission(s)  and  future  investments  can  be  fit  into  this  existing  scorecard  to  ensure  a 
mission  match  exits.  A  record  of  investment  mission  matching  must  be  maintained  with 
investment  records  so  this  match  can  be  evaluated  in  future  reviews. 
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Figure  6.  A  Framework  to  Link  IT  Projects  to  Strategy  After  Ref.  [7] 


3.  Portfolio  Selection  Criteria  Definition 

The  portfolio  selection  criteria  definition  is  the  written  policy  and  procedures  for 
creating  and  modifying  the  minimum  acceptable  attributes  an  IT  investment  must  display 
to  be  considered  for  addition  to  the  agency’s  IT  portfolio.  Developing  an  IT  investment 
portfolio  involves  defining  appropriate  IT  investment  cost,  benefit,  schedule,  and  risk 
(CBSR)  criteria  to  ensure  the  agency’s  strategic  goals,  objectives,  and  missions  will  be 
satisfied  by  the  selected  investments  (Ref  [l]:p.  74). 

The  portfolio  selection  criteria  must  be  distributed  throughout  the  organization  so 
that  the  minimum  expectations  of  potential  investments  are  known  and  understood  by 
managers  and  IT  personnel.  The  portfolio  selection  criteria  definition  is  the  formal 
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declaration  of  the  worst  benefit/cost/schedule/risk  relationship  of  an  investment  that  will 
be  considered  by  the  agency  for  funding. 

Since  the  definition  can  change  if  the  agency’s  priorities  or  the  environment  the 
agency  operates  in  changes,  agency  policy  must  also  determine  at  what  time  interval  the 
selection  criteria  must  be  reviewed  to  verify  it  is  continuing  to  help  the  agency  match  its 
IT  investments  to  its  mission  needs. 

4.  Investment  Analysis 

Investment  analysis  includes  the  data  collected  to  support  CBSR  reports  for 
investments  and  the  use  of  the  data  by  the  IT  investment  board  to  assess  and  rank 
potential  as  well  as  current  investments.  Initially,  for  new  investments,  the  analysis  will 
provide  an  assessment  of  the  agency’s  best  estimate  of  the  future  CBSR  performance  of 
the  investment.  As  an  investment  is  deployed,  the  analysis  will  track  the  realized  costs, 
benefits,  and  risks  so  that  proper  attention  is  applied  to  the  investment  in  the  Control 
phase. 

Agency  policy  must  delineate  who  performs  the  analysis  and  outline  the 
procedure  for  conducting  the  data  collection  and  presentation  to  the  board.  The  policy 
must  also  describe  how  the  IT  investment  board  assesses  the  reports  with  respect  to  the 
portfolio  selection  criteria  and  prioritizes  its  full  portfolio  of  IT  investments. 

The  investment  analysis  process  provides  the  data  that  allows  the  investment 
board  to  select  investments  for  consideration,  rank  the  investments  prior  to  allocation  of 
funds,  and  track  the  performance  measures  of  the  investment  during  its  control  phase. 
The  CBSR  must  be  formalized  so  that  it  can  be  applied  uniformly  to  all  projects  under 
consideration  by  the  agency. 
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The  information  economics  concept  described  in  Chapter  II  is  a  usefiil  way  to 
combine  the  tangible  benefits  and  costs  with  the  intangibles.  Thresholds  are  set  on  the 
scores  from  the  information  economics  to  define  the  portfolio  selection  criteria  definition 
explained  above.  The  CBSR  will  therefore  be  initially  done  for  a  potential  investment  to 
support  its  screening  and  selection,  allow  for  tracking  of  the  investment  in  the  Control 
phase,  and  be  scrutinized  in  the  Evaluation  phase  to  determine  how  the  process  can  be 
improved. 

5.  Proposal  Selection 

The  purpose  of  the  proposal  selection  process  is  to  establish  a  structured  process 
to  select  new  IT  proposals.  It  is  tied  tightly  to  the  portfolio  selection  criteria  definition 
and  investment  analysis  processes  since  these  two  provide  the  data  input  and  minimiim 
requirements  for  new  proposals  to  be  considered  for  development. 

In  order  to  accomplish  the  proposal  selection,  an  individual  manager  should  be 
assigned  responsibility  for  this  process.  This  manager’s  responsibilities  will  include 
ensuring  the  staff  imderstands  the  key  events  in  the  selection  process,  collecting  all  new 
proposals,  and  presenting  all  proposals  umformly  to  the  investment  review  board. 

In  order  to  present  all  proposals  umformly,  there  must  be  a  formalized  procedure 
to  present  the  proposals  in  terms  of  their  mission  match  and  CBSR  reports.  From  this 
presentation  the  investment  board  can  select  investments  and  allocate  funds  based  on 
maximizing  agency  mission  benefits  while  controlling  risk  and  costs.  The  procedure 
must  be  flexible  enough  to  allow  improvement,  but  formal  enough  to  provide  clear 
instructions  to  the  board.  The  output  of  this  selection  process  is  the  recommendations 
from  the  IT  investment  board  as  to  how  the  agency  should  allocate  flmds  and  develop  the 
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agency’s  IT  portfolio.  If  the  process  is  managed  to  properly  address  agency  priorities,  the 
agency  management  decisions  should  rarely  be  different  from  investment  board 
recommendations. 

6.  Portfolio  Development 

The  portfolio  development  process  takes  the  recommended  new  proposals  from 
the  proposal  selection  process,  including  the  agency’s  current  IT  investments,  and 
optimizes  the  IT  portfolio  with  regard  to  mission  accomplishment,  costs,  returns,  and 
risk. 

In  “Assessing  Risks  and  Returns:  A  Guide  for  Evaluating  Federal  Agencies’  IT 
Investment  Decision-making”,  GAO  explains  a  technique  to  display  expenditures  on  IT 
investments  in  various  stages  of  development  [Ref  2].  All  investments  are  assigned  to  a 
portfolio  category.  These  categories  include:  research,  development,  operational,  and 
enhancements.  The  “research”  category  contains  investments  under  consideration  for 
future  use  by  the  agency  and  being  studied  to  gain  a  better  understanding  of  costs  and 
benefits  associated  with  the  investment.  Investments  in  the  “deployment”  category  have 
been  selected  for  use  by  the  agency,  but  are  not  yet  fully  operational.  The  “operational” 
category  contains  the  IT  investments  currently  fully  deployed  and  in  use  by  the  agency. 
The  “enhancements”  category  consists  of  upgrades  to  operational  IT  investments  that 
have  not  yet  been  fully  deployed.  Figure  7  shows  one  way  to  display  agency  IT 
expenditures  by  category,  including  the  percent  breakdown  of  total  expenditures  within 
each  year.  The  investments  should  also  be  displayed  to  connect  them  to  the  agency 
mission(s)  they  address 


35 


fTTTTl 


%i4 


Year  1 

Year  2 

1  Year  3 

Year  4  j 

■Operations 

41% 

45% 

41% 

1  33% 

□  Enhancements 

23% 

1  24%  ; 

48% 

42%  i 

□  Development 

31% 

25% 

9% 

21%  1 

■  Research 

5% 

6%  1 

2% 

4%  j 

Figure  7.  Example  of  IT  Expenditure  by  Category  After  Ref.  [2] 

By  analyzing  the  trend  of  allocations  in  each  category,  the  agency  can  ensure  that 
current  operations  are  not  using  resources  needed  for  future  research  and  develop  projects 
so  that  the  agency  remains  competitive  and  effective  in  IT.  Likewise,  by  relating  the 
projects  in  each  category  to  the  agency  missions  they  support,  the  agency  can  begin  to 
develop  a  succession  plan  for  aging  or  high  cost  investments. 

Figure  8  displays  a  method  of  tracking  the  portion  of  each  category  (operations, 
enhancements,  development,  and  research)  that  is  being  allocated  to  each  mission 
addressed  by  the  IT  portfolio  for  a  particular  year.  Managers  will  be  alerted  to  missions 
that  have  increasing  operational  costs  and  no  research  or  development  on  the  potential 
replacements  when  this  figure  is  expanded  to  show  mission  vs.  category  spending  over 
several  years. 
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Figure  8.  Example  of  Mission  Support  by  IT  Budget  Category  for  One  Year 

7.  Systems  and  Technology  Succession  Management 

This  process  develops  a  schedule  for  the  replacement  of  investments  or  for 
enhancements  to  current  systems  as  a  form  of  technology  refreshment.  The  data 
displayed  in  Figure  8,  if  expanded  over  several  years,  can  be  useful  in  displaying  the 
trends  of  past  spending  and  determining  the  past  lifetimes  of  specific  projects.  The 
lifetime  of  a  project  can  be  driven  by  different  factors  such  as  the  useful  life  of  the 
product  (i.e.,  laser  printers),  suspension  of  support  by  the  vendor,  or  the  rate  of  upgrades 
available  in  the  market  (i.e.,  office  suite  software  or  desktop  computers). 

By  understanding  the  natural  lifetime  of  IT  products,  an  agency  can  plan  research 
and  development  so  that  replacements  or  enhancements  are  budgeted  for  and  ready  for 
deployment  when  they  are  required  to  replace  an  aging  or  outdated  project. 
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8.  IT-Driven  Strategic  Business  Change 

At  the  highest  level  of  maturity,  the  agency  creates  and  maintains  a  knowledge 
base  of  state-of-the-art  IT  products  and  technologies.  From  this  base,  IT  projects  with  the 
ability  to  strategically  change  the  agency  are  identified.  Strategic  changes  to  the  agency’s 
business  practices  are  then  planned  for  and  implemented  based  on  the  identified  IT 
capabilities  in  this  base  of  knowledge. 

This  requires  the  agency  to  constantly  research  cutting  edge  IT  capabilities  that 
may  not  initially  appear  to  “fit”  into  the  agency’s  IT  portfolio.  When  capabilities  are 
identified  that  can  help  the  agency  strategically  change  the  approach  to  a  specific  mission 
or  consolidate  the  efforts  previously  requiring  multiple  investments,  the  new  technology 
is  developed  and  deployed. 

C.  THE  CONTROL  PHASE 

1.  IT  Asset  Tracking 

The  most  important  process  of  the  Control  phase  is  tracking  IT  assets.  The  IT 
investments  must  be  tracked  by  an  inventory  that  is  maintained  in  accordance  with  a 
formal  agency  policy.  The  inventory  consists  of  a  record  of  investments  that  have  been 
selected  as  well  as  a  record  of  the  potential  investments  that  were  rejected  or  deferred. 
Included  in  the  inventoiy  records  are  a  history  of  each  investment’s  current  CBSR  report 
and  the  expectations  of  the  investment,  including  any  history  of  changes  to  the 
expectations. 

The  inventory  record  provides  a  common  location  for  the  documentation  needed 
by  managers  and  the  IT  investment  board  for  various  processes  in  the  Selection  and 
Control  phases.  The  historical  records  of  past  investments  provide  a  database  that  can  be 
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accessed  to  assist  in  assessments  and  future  investment  selections.  The  inventory  records 
of  this  process  are  required  by  the  IT  investment  board  to  complete  a  thorough  selection 
process  ensuring  all  investments,  potential  and  current,  are  considered. 

2.  IT  Project  Oversight 

IT  project  oversight  combined  with  portfolio  performance  oversight  is  at  the 
center  of  the  Control  phase.  The  IT  project  oversight  process  maintains  the  most  current 
cost  and  schedule  data  for  each  investment  in  the  asset  inventory  records  for  review  by 
project  managers  and  the  investment  review  board. 

The  comparison  of  current  cost  and  schedule  with  the  expected  results  is  the  basic 
tool  the  managers  and  IT  investment  board  should  use  to  track  investments  under 
development.  This  will  be  further  enhanced  in  the  portfolio  performance  oversight 
process  when  the  realized  benefits  and  risk  management  are  taken  into  account. 

During  periodic  review  of  projects  by  the  IT  investment  board,  investments  not 
meeting  cost  and/or  schedule  expectations  will  imdergo  further  review.  The  deeper 
review  will  examine  any  corrective  actions  taken  by  project  managers  and  determine  if 
the  investment  should  be  modified  or  cancelled.  The  documentation  of  the  project 
managers’  reviews  and  corrective  actions,  as  Well  as  those  by  the  IT  investment  board, 
are  critical  to  the  IT  management  process  so  that  lessons  can  be  learned  from  the 
problems  and  successes  encountered.  This  history  must  be  maintained  in  the  IT  asset 
tracking  records. 

3.  Portfolio  Performance  Oversight 

The  process  of  portfolio  performance  oversight  enhances  IT  project  oversight  by 
adding  the  benefits  and  risks  of  the  projects  to  the  cost  and  schedule  comparisons.  The 
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benefits  and  risks  include  many  intangibles  that  can  be  addressed  with  the  information 
economics  concept  addressed  in  Chapter  II.  In  order  for  the  uniform  and  repeatable 
application  of  information  economics,  agency  policy  must  be  very  specific  about  the 
measurements  or  assessments  made  and  the  score  associated  with  each.  This  will  vary 
between  agencies  and  will  change  over  time  in  any  agency  as  its  missions  or  tasks  change 
with  time. 

Like  the  project  oversight  process,  investments  not  meeting  CBSR  expectations 
are  identified  for  deeper  investigation.  The  investigations  should  determine  if  any 
corrective  action  is  required,  but  should  focus  on  determining  the  root  cause  for  not 
meeting  expectations  and  produce  an  actions  plan  for  the  recovery  or  termination  of  the 
investment.  Other  possibilities  are  cancellation  or  the  adjustment  of  investment 
expectations  should  earlier  expectations  be  found  unattainable  or  too  costly  and 
investment  cancellation  would  be  more  damaging  to  agency  mission  accomplishment. 
The  reports  and  records  fi-om  these  investigations  must  be  maintained  with  the  asset 
inventory  for  ease  of  review  and  evaluation. 

D.  THE  EVALUATE  PHASE 

1.  Post-implementation  Reviews  (PIR) 

The  Post-implementation  Reviews  (PIR)  is  the  foundation  of  the  Evaluate  phase. 
The  PIR  is  the  process  by  which  an  investment’s  performance  is  reviewed  after  it  is  fully 
deployed  or  cancelled.  The  IT  investment  board  determines  which  investments  require  a 
PIR  and  when  the  PIR  should  be  conducted. 

Agency  policy  must  delineate  who  may  conduct  PIRs  and  what  training  is 
required  of  personnel  that  conduct  PIRs.  Personnel  assigned  to  the  project  may  be  able 
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to  more  efficiently  conduct  the  review,  but  the  agency  may  chose  to  have  personnel  not 
directly  involved  in  the  project  conduct  the  PIR  to  minimize  any  bias  in  the  PIR  process. 
However,  the  availability  of  sufficiently  trained  and  knowledgeable  personnel  is  often  a 
problem. 

The  information  required  for  a  PIR  is  both  quantitative  and  qualitative.  When  the 
asset  inventory  process  is  properly  in  place,  the  majority  of  data  required  for  this  review 
will  be  present  there.  In  addition,  the  agency  will  probably  obtain  data  fi'om  surveying 
stakeholders  and  personnel  assigned  to  maintain  the  investment.  This  data  will  be  useful 
in  evaluating  how  the  future  control  of  investments  can  be  improved. 

The  final  output  fi'om  the  PIR  is  a  report  containing:  conclusions  about  the 
investment’s  actual  results;  user  satisfaction;  a  review  of  the  assumptions  made  during 
the  data  analysis  and  project  oversight  processes;  lessons  learned  fiom  the  project  and 
management  of  the  project;  a  review  of  PIRs  fiom  similar  investments  in  the  past;  and 
recommendations  for  the  management  of  future  investments  or  changes  to  the  procedures 
in  the  agency’s  IT  management  policy  to  improve  the  management  process. 

2.  Feedback  from  PIRs 

The  IT  investment  board  reviews  all  PIRs.  The  board  'will  examine  the  lessons 
learned,  the  validity  of  the  assumptions  made  during  the  investment’s  development,  and 
the  recommendations  made  to  determine  how  the  agency’s  selection  and  control  of  IT 
investments  can  be  improved. 

The  feedback  fiom  PIRs  is  the  primary  means  by  which  the  IT  management 
process  is  subjected  to  continual  improvement  and  adapts  to  changes  in  the  agency’s 
goals  or  environment.  By  using  feedback  and  the  lessons  learned  fiom  past  projects. 
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future  investments  can  avoid  suffering  the  same  mistakes  and  the  agency  can  quickly 
determine  when  processes  that  provided  past  successes  are  failing  to  fit  into  the  current 
environment. 

3.  Investment  Process  Benchmarking 

The  ultimate  goal  in  maturity,  Stage  Five,  is  to  understand  how  the  agency’s 
processes  differ  fi*om  the  best  practices  and  develop  a  method  to  attain  the  best  practice. 
In  order  to  accomplish  this,  the  agency  must  develop  a  method  to  measure  baseline  data 
from  their  own  IT  investment  management  processes.  Once  this  data  is  collected, 
external  and  comparable  best  practices  are  benchmarked  using  the  same  method  to 
measure  internal  baseline  data.  By  comparing  the  two  data  sets,  the  agency  then  makes 

improvements  to  the  IT  management  processes  to  approach  the  best  practices  available 
for  survey. 

The  advantage  this  process  provides  is  more  rapid  improvement  in  internal 
processes  when  an  external  organization  with  more  effective  or  efficient  techniques  is 
found.  The  standard  Select/Control/Evaluate  framework  improves  agency  processes  by 
providing  feedback  on  a  trial  and  error  basis.  Investment  process  benchmarking  can 
improve  the  internal  processes  faster  by  providing  the  vision  of  the  ideal  processes  and 
therefore  a  target  to  achieve. 

This  chapter  developed  a  model  of  critical  processes  required  in  each  phase  of  the 
IT  investment  cycle  for  management  of  an  agency’s  IT  investment  portfolio.  The 
processes  were  grouped  together  by  the  phase  of  the  investment  cycle  they  occur  within 
and  a  brief  explanation  of  the  goals  of  each  of  the  processes  was  explained. 
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The  next  chapter  will  examine  how  this  model  can  be  applied  to  IT  management 
within  a  specific  federal  agency  that  is  small  in  size  but  critical  to  the  information 
gathering  for  the  national  security  of  the  U.S. 
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IV.  IT  INVESTMENT  MANAGEMENT  BY  THE  WHITE  HOUSE 

SITUATION  SUPPORT  STAFF 


A.  INTRODUCTION 

A  model  for  evaluating  an  organization’s  IT  management  processes  was  presented 
in  Chapter  III.  This  model  can  be  used  to  examine  the  processes  in  place  in  order  to 
compare  them  to  the  critical  processes  required  for  an  organization  with  a  mature 
management  plan.  This  model  will  be  used  below  to  assess  the  processes  in  use  by  a 
specific  organization,  the  White  House  Situation  Support  Staff  (WHSSS). 

WHSSS  is  tasked  with  the  selection  and  maintenance  of  commxmications  systems 
and  computer  intelligence  systems  for  the  White  House  Situation  Room  (WHSR),  the 
National  Security  Council  (NSC)  staff,  and  other  White  House  offices.  The  mission  of 
the  WHSR  is  to  provide  current  intelligence  and  crisis  support  to  the  President,  his 
National  Security  Advisor,  and  the  NSC  staff.  In  order  to  accomplish  this  mission,  the 
WHSR  is  required  to  gather  all  information  concerning  worldwide  news,  threats,  or 
conflicts,  and  disseminate  this  information  in  a  useable  format  such  that  the  President  and 
the  NSC  always  have  the  data  they  need  to  make  informed  decisions  concerning  the 
position  of  the  United  States  of  America.  To  collect,  manage,  and  distribute  this 
information,  the  WHSR  must  be  outfitted  with  state-of-the-art  information  search, 
collection  and  management  systems.  WHSSS  administratively  reports  to  the  Defense 
Information  Systems  Agency  (DISA)  and  receives  their  budget  authority  via  the  DISA 
budget.  However  WHSSS  operates  quite  independently  from  DISA. 
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Operationally,  WHSSS  reports  to  the  President’s  National  Security  Advisor  and 
through  this  connection,  reaches  its  customer  base  of  NSC  directorates  and  White  House 
offices. 

The  planning,  purchasing,  operation,  and  maintenance  of  WHSSS’s  IT  equipment 
is  managed  by  WHSSS’s  Systems  and  Technical  Planning  Directorate.  The  Systems  and 
Technical  Planning  Directorate  is  made  up  of  about  36  persons,  including  military 
personnel,  civilian  employees,  and  civilian  contractors.  Procurement  funds  account  for 
about  one  third  of  WHSSS’s  annual  budget  of  about  $6  million  dollars  and  when 
combined  with  Operations  and  Maintenance  for  major  projects,  flmding  for  IT  projects 
account  for  over  half  of  the  obligated  fimds. 

The  remainder  of  this  chapter  will  discuss  the  processes  used  by  WHSSS  to  manage  its 
IT  investments  and  compare  this  policy  with  the  model  developed  in  Chapter  III.  The  IT 
investment  cycle  for  the  WHSSS  along  with  the  critical  processes  (described  in  Chapter 
III)  are  displayed  in  Figure  9.  Note  that  the  critical  processes  for  the  WHSSS  are 
somewhat  different  from  those  discussed  in  Chapter  III.  These  will  be  explained  in  the 
subsequent  sections  for  the  Selection,  Control,  and  Evaluate  phases. 
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Figure  9.  IT  Management  Policy  at  WHSSS 


B.  SELECTION  PHASE  PROCESSES 

1.  Introduction 

The  strategic  planning  process  at  the  WHSR  has  been  relatively  informal  due  to 
the  dynamic  mission  of  the  agency.  The  critical  nature  of  the  tasks  and  operations  of  the 
agency  helps  unify  the  persoimel  in  a  common  objective  and  ease  communication  among 
the  various  decision-makers.  The  lack  of  a  formal  strategic  plan  has  led  WHSSS  to  adopt 
an  IT  selection  process  as  described  above  and  driven  by  “technology  refreshments”  or 
“innovations”  instead  of  the  portfolio  approach  described  in  Chapters  II  and  III. 

WHSSS  is  in  the  unique  position  of  having  to  replace  significant  amounts  of 
equipment  during  the  changeover  of  Presidential  administrations  due  to  requirements  of 
the  Presidential  Records  Act  of  1978.  This  law  changed  the  legal  ownership  of 
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Presidential  records  from  private  to  public  as  well  as  set  guidelines  for  the  restricted 
public  access  to  these  records.  The  law  defined  Presidential  records  as: 

Documentary  materials,  or  a  reasonably  segregable  portion  thereof, 
created  or  received  by  the  President,  his  immediate  staff,  or  a  unit  or 
individual  or  the  Executive  Office  of  the  President  whose  function  is  to 
advise  and  assist  the  President,  in  the  course  of  conducting  activities 
which  relate  to  or  have  an  effect  upon  the  carrying  out  of  the 
constitutional,  statutory,  or  other  official  or  ceremonial  duties  of  the 
President  [44  U.S.C.  13  2201(2)]. 

This  requires  the  storage  of  each  paper  and  electronic  record  resulting  from  the 
President  and  Vice  President’s  official  duties  unless  the  Archivist  of  the  United  States 
approves  of  a  specific  item’s  disposal.  The  scope  of  this  law  is  quite  far-reaching. 

In  order  to  address  the  electronic  records  encompassed  by  the  law,  all  forms  of 
electronic  storage  need  to  be  examined.  This  includes  the  hard  drives  and  other  storage 
media  of  all  computers  used  by  the  White  House  and  NSC,  including  the  media  contained 
in  the  servers  of  the  computer  networks  used  by  all  of  the  offices.  While  the  media 
addressed  by  the  law  is  of  an  administrative  nature,  the  systems  affected  include 
administrative  as  well  as  mission  critical  IT  investments. 

WHSSS  uses  this  requirement  along  with  the  operational  experience  of  equipment 
lifetimes  to  synchronize  technology  refreshment  with  administration  changeovers  of  each 
new  president  as  a  form  of  cost  avoidance.  Since  training  must  be  provided  for  incoming 
personnel,  WHSSS  has  avoided  the  cost  of  training  twice  by  combining  the  technology 
refi’eshment  with  the  time  period  of  largest  personnel  turnover.  A  more  intangible  cost, 
but  one  still  avoided,  is  the  additional  maintenance  cost  that  would  be  associated  with 
operating  equipment  for  a  longer  period  of  time  (approaching  the  end  of  its  useful  life) 

which  may  lead  to  failures  due  to  age  and  wear. 
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2.  Systems  and  Technology  Succession  Management 

The  “technology  refreshment  schedule”  is  the  driving  factor  for  WHSSS’s 
Selection  phase.  That  is,  the  selection  process  is  “periodic”  based  on  the  technology 
refreshment  schedule  instead  of  the  budget  year  as  normally  found  in  organizations. 
When  an  IT  investment  is  selected  for  deployment,  its  total  life  cycle  costs  are  budgeted 
in  the  WHSSS  Program  Objectives  Memorandum  and  are  reviewed  annually  during 
WHSSS’s  budget  estimate  submission. 

When  replacements  of  IT  systems  and  hardware  are  scheduled  for  the  next  fiscal 
year,  a  team  with  members  from  each  department  within  the  directorate  is  formed  to 
choose  the  new  system.  Potential  replacements  considered  consist  of  upgrades  to  the 
current  hardware  (e.g.,  sophisticated  workstations)  or  deployment  of  a  completely  new 
system  (e.g.,  document  management  software).  The  team  has  the  authority  to  evaluate 
the  potential  replacements  and  recommend  the  specific  purchase  to  the  director  of  the 
WHSR.  The  team  membership  will  be  slightly  different  for  each  purchase.  This  differs 
fi-om  the  investment  board  concept  introduced  in  Chapter  III.  The  investment  board  is 
charged  with  making  decisions  concerning  the  IT  portfolio  of  an  agency.  WHSSS’s 
teams  merely  select  the  replacement  from  a  list  of  alternatives  for  a  system  when  the 
replacement  is  scheduled  for  technology  refreshment. 

Team  members  include  individuals  from  the  following  five  departments: 
Financial  Management  and  Acquisition,  Operations,  Research  and  Development, 
Maintenance,  and  User  Support.  A  project  manager  is  chosen  from  the  Research  and 
Development  department  to  lead  the  team. 
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The  input  from  the  Research  and  Development  department  includes  presenting 
state-of-the-art  projects  that  have  been  researched  for  use  by  the  agency  and  identified  as 
having  the  potential  to  strategically  change  the  manner  the  agency  accomplishes  the 
mission  being  addressed.  These  are  the  elements  of  the  IT-Driven  Strategic  Business 
Change  Process  and  this  is  how  WHSSS  moves  technologies  from  research  and 
development  to  deployment.  Likewise,  input  from  the  User  Support  Department  brings 
elements  from  the  Business  Needs  Identification  Process  to  the  team  by  addressing 
customer  needs  and  desires.  A  technology  refreshment  schedule  for  the  next  ten  years  is 
published  annually  and  the  out-years  are  evaluated  on  an  annual  basis  or  when  Research 
and  Development  produces  options  that  need  to  be  addressed  due  to  mission  concerns. 

By  using  the  technology  refreshment  schedule  and  making  efforts  to  incorporate 
IT-driven  strategic  business  change,  WHSSS  makes  investment  retirement  decisions 
based  on  a  combination  of  the  technology  replacement  and  implementation  of 
investments  from  the  Research  and  Development  department. 

The  selection  of  the  replacement  system  could  be  enhanced  by  expanding  the 
comparison  beyond  the  measures  of  cost  and  schedule  to  include  measures  of  mission 
benefit  and  risk  analysis.  This  would  require  developing  and  utilizing  an  information 
economics  scorecard  as  described  in  Chapter  II. 

3.  Business  Needs  Identification  for  IT  Projects 

The  goal  of  identifying  the  business  needs  associated  with  IT  projects  is  to  assist 
the  agency  in  identifying  the  performance  measure  to  use  in  ranking  the  potential 
investments  and  track  the  investment  in  the  Control  phase.  WHSSS  uses  input  from  the 
User  Support  department  to  address  the  users  needs  and  desires,  but  no  formal  connection 
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is  made  (as  with  the  balanced  scorecard  technique  described  in  Chapter  III)  to  formally 
describe  the  critical  success  factors  of  WHSSS’s  stakeholder’s  (financial,  customers, 
internal  business,  and  iimovation  and  learning)  perspectives  and  the  performance 
measures  that  can  measure  progress  in  achieving  success. 

4.  Investment  Analysis 

The  analysis  of  new  IT  investments  to  replace  current  systems  is  based  strictly  on 
a  cost-benefit  analysis.  There  was  no  evidence  fi-om  personal  interviews  with  team 
members  involved  in  a  recent  selection  process  that  resource  limits  were  considered. 
Instead,  the  most  current  technologies  weighed  against  user  needs  and  current  hardware 
architecture  (that  would  not  be  replaced)  were  compared  based  solely  on  costs  and 
perceived  needs  of  the  users  and  maintainers.  This  in  no  way  implies  that  fimding  is 
unlimited,  but  that  the  agency  personnel  seem  to  be  very  effective  in  restricting 
investments  to  those  projects  that  are  mission  essential. 

The  selection  process  starts  fi-om  a  needs  definition.  The  needs  definition  is 
derived  fi-om  any  hardware  architecture  requirements  to  support  interoperability  with 
other  WHSSS  hardware  or  hardware  of  external  agencies  and  the  perceived  needs  of  the 
customer.  Several  alternatives  are  identified  and  evaluated  based  on  the  costs  and 
schedule  requirements.  By  following  this  procedure  for  each  system  upgrade  or 
replacement,  WHSSS  avoids  the  task  of  actually  calculating  an  enhanced  ROI  and 
ranking  the  alternatives,  but  still  selects  the  investment  that  would  have  the  highest 
enhanced  ROI,  thereby  meeting  the  intent  behind  the  GAO  guidance. 
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C.  CONTROL  PHASE  PROCESSES 

1.  IT  Project  Oversight 

Once  a  project  is  selected  and  enters  development,  the  project  manager  helps 
develop  the  project’s  budget  and  tracks  the  costs  and  schedule  throughout  the  deployment 
of  the  investment.  WHSSS  has  no  current  provision  to  address  the  benefits  realized  or  to 
determine  how  the  risks  involved  with  the  investment  are  changing  or  managed  as 
addressed  in  the  Portfolio  Performance  Oversight  Process.  The  positive  aspects  of 
tracking  risks  and  benefit  realization  are  especially  useful  to  managers  to  assess  progress 
early  in  development  and  deployment  when  there  is  no  equipment  yet  operational. 

An  example  that  displays  a  weakness  of  addressing  customer  needs  by  support 
staff  input  instead  of  customer  involvement,  as  well  as  the  problem  of  not  tracking 
benefits  expected  and  realized,  was  displayed  during  a  recent  deployment  of  a  software 
investment  two  years  ago.  The  NSC  staff  is  supplied  a  search  engine  to  aid  in  sifting 
through  the  vast  number  of  news  wires  to  assist  in  their  research  for  news  items.  The 
incumbent  software  needed  to  be  replaced  due  to  year  2000  compliance  problems.  The 
replacement  software  was  chosen  to  correct  the  Y2K  compliance  requirement  and 
increased  versatility  was  expected,  but  the  user  interface  differed  greatly  from  the  old 
software.  Following  implementation,  the  new  software  resulted  in  a  flood  of  calls  to  the 
User  Support  Department  and  great  user  resistance  and  resentment  towards  the  new 
investment.  Involvement  of  the  users  in  this  software  replacement  would  have  prevented 
this  bad  experience. 

As  a  result  of  the  above  event,  software  upgrades  or  replacements  are  now 
implemented  only  after  a  “help  manual”  has  been  developed  and  users  are  trained  on  the 
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differences  between  the  old  and  new  systems.  Additionally,  the  User  Support 
Department  deploys  the  investments  in  each  NSC  directorate  separately  and  identifies  a 
properly  trained  staff  member  in  that  directorate  office  as  the  point  of  contact  to  answer 
user  questions  until  no  longer  needed.  This  method  was  used  in  the  recent  deployment  of 
the  Microsoft  Office  2000  suite  and  users  encountered  very  few  problems. 

2.  IT  Asset  Tracking 

All  IT  hardware  inventory  is  tracked  in  a  database  by  WHSSS.  However,  the 
goals  of  the  IT  Asset  Tracking  Process  cannot  be  realized,  because  the  database  merely 
tracks  the  receipt  and  physical  location  of  the  equipment.  There  is  no  method  currently 
available  in  the  database  to  connect  an  entry  for  a  piece  of  equipment  to  a  specific 
investment.  Operators  in  the  Maintenance  and  Operations  Departments  have  developed 
methods  to  track  the  equipment  that  they  regularly  deal  with,  but  this  is  done  in  an  ad  hoc 
manner. 

There  is  a  project  under  review  that  will  replace  the  current  database  with  a 
system  that  will  identify  individual  pieces  of  hardware  with  their  respective  investment 
project  as  well  as  tie  in  lessons  learned  during  the  development  (fi-om  Research  and 
Development  Staff)  and  operation  (fi-om  the  User  Support  Staff).  This  would  greatly 
improve  the  IT  Asset  Tracking  as  well  as  improve  the  feedback  fiom  the  lessons  learned 
in  the  Evaluate  phase. 

3.  Modification  or  Cancellation  of  IT  Projects 

“Continued  investment  analysis”  (cost  and  schedule  only)  is  conducted  by  the 
program  manager  during  IT  investment  deployment  to  track  the  investment’s 
performance.  No  record  of  corrective  actions  taken  is  maintained  during  the  deployment. 
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There  are  no  formal  procedures  to  report  problems  during  the  project’s  deployment  other 
than  the  thresholds  at  which  the  level  of  management  required  to  add  additional  funds  to 
the  investment’s  budget  is  specified. 

The  use  of  the  balanced  scorecard,  as  described  in  Chapter  III,  would  provide 
additional  performance  measures  that  could  be  tracked  and  used  to  determine  when 
corrective  action  and  additional  managerial  attention  is  required. 

D.  EVALUATE  PHASE  PROCESSES 

There  is  currently  no  formal  process  to  perform  Post-Implementation  Reviews  or 
provide  feedback  from  lessons  learned  so  that  future  improvements  can  be  made  to  the 
Selection  and  Control  phases.  Instead,  individual  WHSSS  departments,  especially  the 
Research  and  Development  staff,  have  attempted  to  gain  the  benefits  of  a  Post- 
Implementation  Review  by  developing  a  procedure  to  incorporate  lessons  learned  into  the 
next  Selection  and  Control  review  of  a  given  investment.  This  includes  the  “corporate 
knowledge”  of  individuals  involved  with  the  last  technology  refreshment.  However,  this 
knowledge  is  often  difficult  to  locate  since  these  vital  people  may  not  be  involved  in  the 
next  process  or  they  may  no  longer  be  employed  by  WHSSS. 

An  example  of  how  the  need  for  a  PIR  process  recently  became  evident  was  with 
the  last  replacement  of  laser  printers.  Few  personnel  fi-om  the  previous  purchase  decision 
were  involved  in  the  replacement  process.  The  decision  for  the  replacement  printers  was 
based  mainly  on  the  relationship  with  the  current  vendor  and  a  replacement  was  selected 
fi-om  the  most  technologically  advanced  options  based  on  cost  and  schedule  of  delivery. 
As  a  result  of  the  options,  the  new  printers  had  a  larger  footprint  and  did  not  fit  in  some 
office  locations.  Additionally,  a  technical  difference  existed  in  the  manner  the  printer 
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processed  templates  in  the  current  software  suite.  The  User  Support  Department 
struggled  with  answering  user  questions  and  developing  new  templates.  The  User 
Support  and  program  manager  assigned  with  selecting  laser  printers  now  have  a  long  list 
of  lessons  learned  to  help  in  the  next  selection  process,  but  these  exist  in  individual  files, 
not  in  organizational  records.  The  planned  replacement  for  the  IT  Asset  Tracking  System 
should  help  alleviate  this  problem. 

The  purpose  of  the  Evaluate  phase  is  to  examine  the  lessons  learned  from  each 
specific  project  and  use  these  to  adjust  the  processes  in  the  Selection  and  Control  phases 
to  continually  improve  the  agency’s  management  of  its  IT  investments.  While  WHSSS 
makes  an  effort  to  apply  lessons  learned  to  the  next  decision  process,  there  is  no  process 
in  place  to  modify  the  processes  to  incorporate  these  lessons  learned.  To  address  the 
lessons  learned  issue,  WHSSS  requires  two  changes  to  current  practices.  First,  the 
processes  used  need  to  be  formalized  in  some  manner  so  that  they  are  readily  available 
for  use  and  modification.  Second,  a  PIR  process  needs  to  be  formalized  to  allow  the 
lessons  learned  to  improve  the  agency’s  processes  and  make  the  lessons  learned  more 
accessible.  The  use  of  the  PIR  would  also  provide  a  better  method  of  assessing  customer 
satisfaction  with  the  investments  instead  of  relying  on  User  Support  department  records 
of  trouble  calls. 

This  chapter  has  used  the  IT  investment  management  concepts  from  earlier 
chapters  to  evaluate  the  one  specific  federal  agency’s  practices.  It  appears  that  this 
agency,  through  a  combination  of  informal  and  formal  practices  does  comply  with  the 
spirit  of  the  Clinger-Cohen  Act’s  intent.  That  is,  to  assess  the  effectiveness  of  IT 
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management  processes.  In  addition,  WHSSS  personnel  are  already  making  changes  and 
taking  action  to  improve  those  IT  management  practices  that  fall  short  of  the  objective. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

1.  Evaluating  IT  Management 

By  the  passage  of  the  Clinger-Cohen  Act  and  the  active  involvement  of  the 
General  Accounting  Office,  an  oversight  arm  of  the  Congress,  in  IT  issues,  the  U.S. 
Congress  has  signaled  the  federal  agencies  that  management  practices  for  IT  investments 
must  improve.  This  is  understandable  given  the  ever-increasing  expenditures  on  IT 
projects  vvithin  the  federal  government.  This  thesis  has  attempted  to  address  the  major 
concerns  of  both  the  Congress  and  the  managers  responsible  for  implementing  new  IT 
management  practices. 

To  build  a  basic  outline  for  an  agency’s  IT  investment  management  policy,  this 
research  first  utilized  the  GAO’s  three  phases  of  the  IT  investment  cycle  as  the  major 
themes  for  the  model.  These  phases  are  Select,  Control,  and  Evaluate.  In  addition,  the 
critical  processes  outlined  by  GAO  in  the  maturity  stage  model  (displayed  in  Figure  3) 
provided  new  concepts  for  addressing  questions  of  efficiency  and  effectiveness  within 
each  of  the  IT  investment  cycle  phases.  The  maturity  stages  was  an  important  addition 
because  it  sharpens  the  focus  on  an  organization  that  is  positioned  at  a  specific  level  of  IT 
management,  ranging  from  simple  project  oriented  organizations  to  those  in  which  IT  is 
an  integral  part  of  its  strategy  and  mission. . 

The  complete  model  was  discussed  in  Chapter  III,  including  the  critical  processes 
required  by  each  phase  of  the  investment  cycle.  Then  in  Chapter  IV  the  recommended 
IT  investment  management  model  was  used  as  the  basis  for  comparing  the  IT 
management  processes  that  were  in  place  in  a  specific  agency  (displayed  in  Figure  9). 
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This  management  tool  helped  identify  the  differences  between  the  agency’s  processes  in 
use  and  those  of  the  ideal  agency  with  fully  mature  IT  investment  management  processes. 
The  major  contribution  of  the  model  is  that  it  allows  an  agency  to  quickly  assess  what 
improvements  need  to  be  made  to  its  processes  and  what  new  processes  should  be 
investigated  for  proper  fit  in  the  agency’s  business  environment  and  possible 
implementation. 

At  the  highest  maturity  level  of  IT  technology  awareness,  the  IT-Driven  Strategic 
Business  Change  process  is  the  vision  of  using  IT  to  change  the  focus  or  direction  of  an 
agency  to  gain  strategic  advantage.  This  is  one  of  the  goals  of  CCA  and  these  gains  can 
be  realized  on  the  battlefield  or  in  the  business  operations  of  the  government. 

When  developing  IT  investment  management  processes  for  an  agency  or 
assessing  the  processes  in  use  one  essential  point  must  be  addressed.  While  there  is  a 
common  foimdation  on  which  to  build  an  agency’s  IT  management  policies,  each  agency 
needs  to  evaluate  the  specific  operations  they  face  and  choose  the  implementations  that 
best  fits  their  agency  and  the  environment  they  operate  in.  This  is  both  recognized  by 
GAO  [Ref.  1  :pp.4,  5]  and  displayed  in  the  example  of  the  White  House  Situation  Support 
Staff  in  Chapter  IV. 

2.  WHSSS  Approach  to  IT  Management  Processes 

The  organization  selected  as  a  site  for  applying  the  recommended  IT  management 
model  was  the  White  House  Situation  Support  Staff  While  very  small  in  comparison  to 
other  federal  agencies,  this  organization  can  be  characterized  as  IT  intensive.  That  is,  the 
critical  mission  of  the  agency  in  U.S.  national  security  matters  requires  use  of  the  best 
information  technology  and  communication  systems. 
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As  pointed  out  in  Chapter  IV,  WHSSS  has  chosen  a  management  technique  that 
balances  the  requirements  of  periodic  technology  refreshments  with  that  of  purchases 
associated  with  large  personnel  turnovers  and  legal  mandates.  Presidential  elections 
force  personnel  turnovers,  which  obviously  are  beyond  the  control  of  the  WHSSS. 
Likewise,  legal  archive  requirements  place  a  heavy  burden  on  IT  for  document 
management  and  storage.  The  technology  succession  plan  they  operate  under  assumes 
cost  avoidance.  Also,  they  manage  the  upgrade  or  replacement  of  systems  to  attempt  to 
choose  the  system  with  the  best  return  on  investment  without  actually  calculating  a  ROI. 

Without  documenting  the  procedures  they  use,  or  starting  from  the  requirements 
of  CCA,  the  WHSSS  have  successfully  used  their  own  procedures  to  meet  the  intent  of 
the  CCA  by  striving  to  choose  the  product  with  a  cost  and  lifetime  that  best  fits  the 
succession  schedule  that  has  been  chosen  for  the  environment  they  work  within.  Without 
documentation  of  the  procedures  used,  the  data  used  in  decision-making,  or  a  history  of 
past  decisions  made,  it  is  difficult  to  predict  the  repeatability,  completeness,  and 
efficiency  of  future  IT  investment  cycles  or  explain  past  cycles. 

The  cost  avoidance  theory  of  their  current  IT  management  technique  along  with 
the  technology  refreshment  process  that  evaluates  new  abilities  of  maturing  technologies 
and  addresses  the  customer  needs  and  desires  may  indeed  be  more  efficient  than 
developing  the  portfolio  approach  outlined  in  Chapter  II.  However,  without  a  list  of 
strategic  goals  and  the  connection  of  these  to  the  elements  of  the  Balanced  Scorecard  in 
Chapter  III  it  is  very  difficult  for  WHSSS  to  develop  performance  measures  to  track  the 
mission  accomplishment  provided  by  IT  investments.  Also,  it  is  difficult  to  defend  the 
current  technique  by  comparing  it  to  GAO  and  OMB’s  best  practices. 
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Once  a  strategic  plan  is  developed  and  implemented,  the  current  processes  can  be 
evaluated  further  and  either  be  improved  or  replaced  with  a  more  efficient  management 
technique. 

B.  RECOMMENDATIONS 

1.  Evaluating  IT  Management 

As  noted  in  Chapter  III,  the  maturity  levels  identified  by  GAO  may  be  difficult  to 
apply  to  an  agency  that  has  IT  projects  using  processes  in  some  or  all  of  the  maturity 
levels,  but  missing  critical  processes  at  the  lower  maturity  stages.  This  would  be 
expected  for  most  agencies  that  have  been  managing  IT  projects  in  the  past,  and  would 
result  in  a  low  maturity  assessment  despite  the  higher  level  processes  in  use.  Using  the 
model  developed  in  Chapter  III  to  address  all  the  critical  processes  and  measure  the 
effectiveness  of  each  process  can  be  a  much  more  useful  technique  to  assess  the  IT 
management  of  an  agency  and  allow  them  to  address  the  weaknesses  present  in  all  phases 
of  the  investment  cycle  at  one  time. 

As  mentioned  in  Chapter  III,  the  efforts  in  the  Department  of  the  Navy  to  develop 
a  information  scorecard  for  screening  and  ranking  potential  investments  is  helpful  for 
agencies  in  the  Navy,  but  the  guidance  should  be  expanded  to  explain  how  the  scorecard 
can  be  altered  to  meet  a  specific  agency’s  mission  needs  and  business  environment.  This 
identifies  two  very  important  points;  guidance  must  be  given  concerning  how  to  screen 
and  rank  investments,  but  this  guidance  must  be  generic  enough  to  apply  to  all  agencies 
in  the  department  and  the  guidance  must  include  a  description  of  how  an  agency  can  alter 
the  scorecard  to  best  fit  the  specific  agency. 
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2. 


WHSSS  IT  Management 


The  vision  of  the  Clinger-Cohen  Act  is  to  take  the  benefits  of  the  Government 
Results  and  Performance  Act  and  extend  them  to  the  IT  investments  of  the  government. 
Without  a  strategic  plan  and  the  focus  of  the  organization’s  direction  on  the  goals  set  in 
the  plan,  it  is  difficult  to  measure  the  efficiency  by  which  WHSSS  manages  its  IT 
investments.  Once  a  strategic  plan  is  developed,  WHSSS  can  use  the  balanced  scorecard 
outlined  in  Chapter  III  to  connect  its  mission  to  the  critical  success  factors.  When  this 
connection  is  made,  performance  measures  can  be  chosen  to  track  the  mission 
accomplishment  of  its  IT  investments  purchased  to  address  these  missions.  Additionally, 
the  use  of  performance  measures  will  provide  WHSSS  with  a  method  to  measure  and 
display  the  improvement  in  mission  accomplishment  with  each  generation  of  technology 
refreshment. 

While  many  effective,  and  in  some  cases  efficient,  processes  exist  in  WHSSS  IT 
management,  the  most  obvious  flaw  is  that  neither  documentation  of  the  techniques  used 
in  each  process  nor  the  data  from  previous  decisions  (including  the  decision  made  and  the 
data  used  to  make  that  decision)  is  readily  available.  Without  a  documentation  of  how 
these  processes  are  conducted  and  history  of  the  decisions  made,  the  repeatability  and 
completeness  of  the  processes  are  difficult  to  predict  since  they  depend  on  the  presence  of 
personnel  knowledgeable  of  previous  decisions.  Likewise,  the  efficiency  of  future 
decisions  cannot  be  predicted  and  the  efficiency  of  past  decisions  are  difficult  to  evaluate 
without  a  record  of  the  data  and  techniques  used  to  make  the  decision 

The  most  obvious  improvement  that  can  be  made  to  WHSSS  IT  investment 
managements  is  displayed  in  the  empty  block  in  the  Evaluation  phase  portion  of  Figure  9. 
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Without  a  fonnal  Post  Iinpleinentation  Review  process  and  a  method  for  the  review  to 
provide  feedback  to  improve  future  Selection  and  Control  processes,  WHSSS  fails  to 
address  the  second  requirement  of  CCA.  The  processes  in  use  must  be  continually 
improved  to  ensure  inefficiencies  of  the  past  are  discovered  and  prevented  from  occurring 
in  the  future. 

C.  AREAS  FOR  FURTHER  RESEARCH 

Two  areas  of  research  could  improve  the  effectiveness  of  evaluating  IT 
management  processes.  First,  the  maturity  level  model  outlined  by  GAO  may  be 
improved  by  developing  a  model  whose  maturity  levels  address  all  phases  of  the 
investment  cycle  (Select,  Control,  Evaluate)  and  each  maturity  stage.  For  instance.  Stage 
Two  would  consist  of  the  processes  necessary  for  simple  application  of  Select,  Control 
and  Evaluate  processes.  All  three  phases  would  then  gain  more  mature  processes  as  the 
agency  matured  through  each  stage.  This  would  be  more  useful  than  the  current  model 
since  agencies  generally  have  processes  in  place  that  exist  in  multiple  maturity  stages  of 
the  GAO  model.  This  newer  model  would  allow  an  agency  to  improve  its  Selection 
phase  processes  in  Stage  Two,  while  also  improving  its  Control  phase  processes  that  may 
be  at  Stage  Four. 

Second,  there  should  be  more  research  into  the  methods  used  by  both  corporations 
and  government  agencies  to  link  IT  investments  to  missions  and  techniques  such  as  the 
balanced  scorecard  technique  described  in  Chapter  III.  This  would  assist  in  the  selection 
of  performance  measures  for  tracking  IT  investment  performance,  and  help  improve  the 
Selection  phase  ranking  and  the  methods  used  to  track  investments  in  the  Control  phase. 
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Additionally,  WHSSS  is  a  very  unique  agency  in  terms  of  the  enviroimient  it 
operates  in.  This  somewhat  limited  the  application  of  the  model  developed  in  Chapter 
III.  Additional  research  that  applies  this  model  to  other  agencies,  especially  larger 
organizations  with  numerous  divisions  with  different  IT  needs,  is  required  to  better  test 
its  ability  to  assess  an  agency’s  IT  management  regardless  of  size  and  IT  project 
complexity. 

Concerning  IT  investment  management  at  WHSSS,  once  the  strategic  plan  has 
been  developed  and  implemented,  a  study  of  the  improvements,  if  any,  that  were  made  to 
the  agency’s  IT  management  plan  should  be  conducted.  This  can  be  based  on  the 
research  of  this  thesis  or  based  on  the  focus  provided  by  the  strategic  plan.  Such  a  study 
will  aid  WHSSS  in  both  setting  the  baseline  of  their  investment  processes  and 
implementing  investment  process  benchmarking. 
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